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ABSTBACT 

The present stud; vas designed to assesfs the effects 
of a combination of feedback and personal consultations using college 
student evaluations on faculty performance. Student evaluation 
feedback and personal consultations were conducted at least a- 
semester before any follow-up data vas gathered. The results indicate 
that providing computerized results of college student evaluations 
along with individual faculty consulting sessions helped the 
instructors significantly improve their student ratings on two 
instructional dimensions. (Author) 



/\b8traet 

The present etudy vaa designed to asseed the effecta of a coabination 
of feedback and peraonal conaultatlona using college atudent eroluatioiui 
ott faculty perfottaance. Student evaluation feedback and peraonal conaulta«* 
tiona vere conducted at least a aemster befoice any follov-iq^ data^ae 
gathered. Hhe results indicate that providing coiqtuterised results of 
college fitudent evaluations along with individual faculty consulting 
sessioTiS helped the instructors significantly issprove tb^ir atud^t ratings 
on two instructional diiaenslons. 



BEST copy AVAiLAB.LE 



TOE ITSEFDLNESS OF STUDENT EVALUATIOKS IK 
IMPROVING COLLEGE TEACHING 
Lawrence M« Aleaisonl^ 
In the past few yeara, as a result of the 1970 student striices aid the 
esaphasis on accountability, course and instructor evaluation has been placed 
in the spotliglit. In an atteiopt to build a total instructional evaluation 
systea, a great deal of etqphasls has been placed on student evaluations of 
course and instructor and the fact that such evaluations oust be both feltoblLa 
and valid to be considered an integral part of a total instructional evaluation 
system. Costln, Greenough, and Menges (1971) after extensive reviews of 
existing instruasents concluded that student opinions do affect leaning and 
provide a source of quite reliable and valid data relative to the effective- 
ness of Instruction. A question that naturally arises in developing such 
evaluation systems is "Can student evaluations of instruction and IntBtructor 
be useful In itaproving college teaching once they are loade available to the 
Instructor?" 

Althou^ there has been a great deal of anecdotal evidence from instructors 
and researchers to suggest that atudent evaluations do have a positive effect, 
very few studies are available that deal with that effect on college level 
instruction. In two fairly recent studies by Miller (1971) and Centra (1973) 
student evaluations gathered at sdLdseo&ep.ter and presented to the participating 
instructors as feedback were compared to the endf-of-seiisester ratings « Miller ^e 
study ev^loyed one college's graduatei student teaching assistants whereas 
Centrals study employed five different small colleges^' teaching faculty* 



^lie author is indebted to Nancy Halff for h(sr valuable editorial and 
statistical asslstance»and to Pamela Z. Hexner and Jeffrey A* Slinde in 
conducting the data analysis* 



In both studies no significant differences were fotmd between the mldseo^ster 
and ehd-^of-semester student ratings of Instruction. 

A more recent study by Braunsteln, Klein, and Pachla (1973) atteii?)ted 
to assess the effects of college student evaluations on faculty performance 
by providing feedback midway through the semester at one college. A 
comparison of the shift in median student ratings between the midsemester 
and end-of-semester (seven to eight weeks later) indicated significantly 
greater increiaents for the e3q)erieental feedback group. This study appears 
to be the only one available yielding positive effects of student evaluation 
feedback at the college lev3l. The authors do^ however^ point out that the 
data used should be viewed with caution because the measures used were in 
the form of shifts in the class medians of one scale point or more. 

Of the various systems developed for student evaluation of course and 
instructor, the Illinois Course Evalu^itlon Questionnaire (CEQ) has perhaps 
the most extensive reliability^ validity, and norm base data fo^^^ 
Normative data have been collected conitlnuously since 1966 by the Measurement 
and Research Division of the Office of Instructional Resources at the 
University of Illinois, Urb ana-Champaign camptis. The CEQ is used to collect 
student attitudes toward a course and instructor, and its purpose is to 
enable faculty members to collect evaluative information about their teaching. 
As the ntisiber of measures on each course is Increased, it becomes possible to 
obtain a relatively stable indication of the difference between courses. 
This aids in the interpretation of the actual differences between an obtained 
section score for a particular instructor and the average scores for all the 
sections represented in that course* 



The analysis of item Intidr-relationships and the subscore Inter-relation-* 
ships Indicated that no one element » related to a course, dlspropcrtionately 
Infl^i^ced the students* evalvatlon of the course (Aleamoni & Spencer » 1973)* 
HoHevrr, it appears that there also is a "general course attitude" cultivated 
by the student as he is exposed to previous students* comments, the instructor, 
the textbook, the course, etc., and this is the framevork from which he 
responds when answering the CEQ items. 

It would seem on the basis of three vali^jLty studies (Stalllngs & 
Spencer, 1967; Swanson & Sisson^ 1971; and Aleamoni & Yimer» 1973)9 the face 
validity of the CEQ^ and Its high reliability, that extremely low scores 
on a particular subscore do indicate problem areas in an instructor's 
teaching program, whereas, stable high scores point to an effective In&truo- 
tlonal program as viewed by students. All available validating evidence 
to date (both published and unpublished studies), Indicates that the CEQ 
does indeed identify courses that are conside^^^ or poor* 

After using the CEQ, the Instructor receives results (see Appendix A) 
which allow him to compare his course item me;an8 to institutional course 
item means (via deciles) and to covapaxe his course stibscale means to norm 
stbscale means categorized by; (a) rank of instructor, (b) level of course, 
(c) institution, (d) college, and (e) all Institutions that have used the 
CBQ throughout the Ihiited States. The subscale results allow the instructor 
to obtain an indication of major areas of strengths and weaknesses in the 
course. Once the areas of weakness have been identified by the subscales, 
then looking at the item results helps to focxis on the more specific problem 
^reas. Tlie CEQ items are not coiq>letely diagnostic but do serve to elicit 
diagpostia responses from the instructor teaching the course « It provides 
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a maana whereby some self^-evaluation of the teaching process can occur; other 
laeans cm be arranged and are available such as asking more diagnostic questions 
In the optional Item section avallabla on the CEQ forms » (see Appendix B) or 
having peers sit In on actual class sessions , etc. 

The Optional Item Catalog (Aleamonl & Brandenburg, 1973) was generated 
In order to provide Instructors with Items that may be more relevant or 
diagnostic for their particular courses. 

After the Instructor has decided to use the CEQ and/or any optional 
Items of hlB choice. It Is then up to him to decide what to do with the 
data. If he feels that the Interpretation manual (Aleamonl, 1972) and 
abbreviated Interpretation sheets are not sufficient to help him Identify 
areas that may need Improvement In the course, he can then arrange for a 
ccmference with one of the members of the Measurement and Research Division 
staff. 

It has been throu^ a process such as this that Instructors have been 
able to use student evaluations to Identify Instructional problems and then 
rectify them. Obviously, the success or failure of such a venture rests 
solely with the Instructor and his willingness to both gather and use the 
data provided him. 

Since the author has been involved in utilizing student evaluations to 
help Instructors identify and diagnose Instructional problems, the data 
was available to conduct a study on the effects of feedback in the following 
year. The present study, therefore, provides data beyond mldsemester to 
end-*of*-8eme8ter comparisons and also looks at the effect of feedback 
consultations . 
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Method 

Deeaription of Suboects 

Instructors at two different institutions (University of Arizona at 
Tucson and Sheridan College at Sheridan, Wyoming) who had used the GEQ during 
the fall 1971 and spring 1972 terms for their courses were the sitbjects of 
the present study. The experimental group consisted of 20 instructors 
teaching 34 courses at the University of Arizona and 12 instructors teaching 
16 courses at Sheridan College. The control group consisted of 13 instructors 
teaching 18 courses at the University of Arizona and 2 Instructors teaching 
2 courses at Sheridan College. There were 2,980 students In the experimental 
group of 50 courses and 874 students in the 20 control group courses* 

Bach of these Instructors was scheduled to talk with the author about 
his/her results. The conferences were conducted lndlvldua].ly at the home 
caucus of the Instructor and took approximately 15 to 20 minutes. The 
conference began with a close scrutiny of the CEQ subscale results to see If 
• any problems existed based on the norm data available. If a problem area was 
identified (such as Method of Instruction) then a close look at the Items 
making up that subscale would be in order. If in the discussion with the 
Instructor the source. of difficulty was identified^ then the discussion shifted 
to possible ways of trying to resolve the difficulty. If» on the other hand^ 
the source of difficulty was not identified u»lng the existing items and the 
instructor's recall, then other procedures (such as the use of optional 
items, classroom visitation, video-taping, etc.) were explored to be able 
to identify the specific problem. 

In assessing the usefulness of student evaluations in improving college 
teaching, each instructor who had participated in the individual conferences 

ERLC 



was subseqiiently followed up to see if any significant change had occurred in 
their student ratings in subsequent terms (fall 1972) In the same couraeCs). 

Since a majority of the instructors in the experimental group had 
decile ratings below eight or nine (the criterion used to determine problem 
areas) on the Instructor subscale and possibly one or more of the other four 
siibscales of the CEQ, only those instructors with decile ratings of zero 
through seven on the Instructor subscale were used to make up the experimental 
group of the present study. 

Instructors who were unable to participate in the individual confexences 
due to class conflicts or previous commitments were used to make up the 
control group. These instructors had used the CEQ at the saxsa time as the 
experimental group. It should be emphasized that these instructors were all 
planning to participate In the individual conferences and personally expressed 
their disappointment to the author and their department head In not being-able::^ 
to meet on the scheduled days. 

All CEQ data was gathered at or near the end of each semester. 

A two-way repeated measures analysis of variance (ANOVA) design was 
used to analyze the data. 

Restilts 

Means, standard deviations^ class sizes, and norm deciles were obtained 
for each of the above instructors on five of the CEQ subscales as well as the 
Total and are presented in Table 1. The data on the five subscales was then 
analyzed to determine if the conferences had any olgnlflcant effect on helping 
the instructor improve his/her teaching as reflected in subsequent student 
evaluations measured by the subscales ox the CEQ. 
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Experimental 
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Institution 
Instructor 


Arizona 5 
Arizona 6 
Arizona 7 
Arizona 8 
Arizona 9 
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Inspection of the data in Table 1 indicated that the two-*per8ca*control 
group from Sheridan College was not a representative aaiz^le. The entire 
Sherldaa sample was then deleted and the analysis focused on only the 
University of Arizona data. 

The ANOVA results for the two subscales (Course Content and Instructor) 
that yielded significant F-ratlos are presented In Table 2. The resiilts 
presented In Table 2 Indicate that there was a significant Interaction between 
the pre-post and experliaental->control student ratings of the Course Content and 
Instructor subscales. Upon inspection of the plot of the significant 
Interactions (see Figures 1 and 2) it was obvious that slti:ple main effects 
tests of the interactions were called for in order to detensine whether or not 
significant laean rating differences had occurred between the pre and post 
oBdSures. 

Table 2 

ANOVA Results for the Course Content and 
Instructor Subscales of the CEQ 

Source I df { SS I MS | F (Probability 



Course Content 



Pre-Post 

Pre-Post X Experimental- 
Control 


1, 40 
1, 40 


.05606 
.13395 


.05606 
.13395 


3.0889 
7.3809 


.0865 
.0097 


Subjects X Pre-Post 


AO 


.72594 


.01815 






Instructor 


Pre-Post 

Pre-Post X Experimental- 
Control 


1» 40 
1, 40 


.01263 
.16663 


.01263 
.16663 


.6000 
7.9159 


.4431 
.0076 


Swjects X Pre-Post 


40 


.84199 


.02105 
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Figure 2. The Pre-Post by Experimnntal-Control Interaction for the 
Instructor Subscale 
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The restilts of those slisple loala effects tests (presented In Tables 3 
and 4) indicate that there was a significant Increase in the pre-*post 
oieasures for the experltoental group on both subscales, but no significant 
decrease In the pre-post tneasures for the control grotq>. This suggests that 
the CEQ results and the Individual conferences had an effect on the Instructor 
for both the Course Content and Instructor subsealest 

Table 3 

Siiople Main Effects Tests for the Pre-Post x Experimental-Control 
Interaction on the Course Content Siibscale of the CEQ 



Source 


df 


t 


Probability 


Experimental 


Pre-Post 


40 


3.085 


p <..01 


Control 


Pre-Post 40 


.911 


p < .40 


Pre 


Expe rlmen tal-Cpn t rol 


40 


.726 


P < .50 


Post 


Experimental-Control 


40 


1.690 


p < .10 
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Table 4 

Sinple Main Effects Tests for the Pre-Post x Esperlmental-Control 
Interaction on the Instructor Subscale of the CEQ 



Source 


df 


t 


Probability 


EzperliDeatal 


Pre-Post 


40 


2.625 


p < .02 

r 


Control 


Pre-Post 


40 


1.653 


p < .20 


Pre 


ExperlineataI*^Con tro I 


40 


.604 


p > .50 


Post 


Eiq;>erltiien t aI«-Con t to 1 


40 


1.760 


p < .10 



An Inspection of the norm decile shifts for the two subacaleo (presented 
in Tables 5 and 6) singly reinforces the data presented in Figures 1 and 2. 
The average norm decile increase or decrease on the Course Content subscale 
for the e3q>eriiDental group was 1.13 coiqpared to **.2d for the control group 
and on the Instructor subscale, the experimental £,roup 1.17 coiapared 
to -.67 for the control group. The range of norm decile increase or 
decrease on the Course Contetit subscale for the experimental group is from 
-2 to 4 coxspared to from -3 to 3 for the control group and on the Instructor 
subscale, the experimental group ?^as from •*4 to 5 compared to from ^7 to 3 
for the control group. 
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Nona Decile Changes for the Course Content Subscale 
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0 
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33 
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Keen 
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Table 6 

Horn Dedls Ch^tnges for the Inatnictor Sv&scale 



Instructor 


, E]q>erlaental 


Control 


Pre 


Post 


Increase 
Decrease 


Pre 


Post 


Increase 
Decrease 


Arlzcna 


1 


2 


7 


5 


0 


0 


0 


2 
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2 




4 \ 


-1 
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-1 
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7 


9 


2 
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2 


8 
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6 


6 


2 


9 


9 


0 


10 
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6 
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8 


4 


-4 


11 
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7 
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-2 


12 
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-7 


13 
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1 


14 
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-4 
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-2 
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16 
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7 
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19 
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-3 








20 
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1 








21 


7 


8 


1 








22 
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0 








23 


1 


0 


-1 








24 


7 












28 


7 


7 


0 








33 


6 


6 


0 








34 


7 


8 


1 








Mean 


5.33 


6.50 


1.17 


5.61 


4.94 


-.67 
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Discussion 

The results of the present study indicate that providing CEQ coiBputerlJted 
results ss veil ub conducting individual consulting sessions with each 
instructor vho received a normative rating indicating areas in need of 
izBproveaent, helped the instructors to improve their rating a scaaestar 
or y*5ar later in the sai&e course. This indicates that instructors who are 
provided with such Information er^i the opportunity to diccuss and explore 
possible ways of rectifying their areaij of instructional weakness can 
improve their instruction if they choose to do so* 

The resofcts of the present study further suggest that instructors (when 
provided with the appropriate information and an oppi^rtunity to interact 
with ipo&aone concerning that information) can significantly improva their 
course content and their personal classroom techniques during one academic 
year. It should also be noted that the magnitude of the improvement in each 
of the alcove areas was not the same and, therefore^ one should not expect 
li^provemftnt In one area to necessarily result in improvement in any of 
the others without some purposeful effort on the part of the instructor. 

Even though the interaction effects indicated that a larg«)r difference 
vvtwe^ pr^poat ssasurss e:d.cted for the essperissit*! V€i^u* ciontrol 
group, there were no significant differences between thi cOQtrol and 
experimantal groups on any of the six dependent m^asui^rs. One possible 
explanstlon for the lack of significant differences is the way the control 
group was actually selected. If the instructors could have been assigned 
randomly to the experimental and control groins, aignlf leant differences 
^l^t have been possible. 
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The present sCudy coafinas the anecdotal evidence as veil as the one 
study by Braunsteln, Klein end Pachla (1973) that feedback to college 
instructors cm lead to significant inprovenent in their instruction, 
Bovever, it should be pointed out that all the studies cited basically 
looked at the effect of feedback vithin a semester or ter& and not betveca 
seoesters at vas the case with the present study. Also, no atteopt vas 
aade in those studies to provide Instructors vith the opportunity to discuss 
the results of student evaluations vith an independent resource person yft^ 
could help thsQ diagnose their instructional prx^bler^. 

It appears 9 therefore, that the co9i>inatlon of feedback and personal 
consultation on student evaluations of course and inatnjtctor provide a 
better fraaswrk for encouraging an instructor to use such results for 
potential Inprove^aent vhen cohered to feedback of results alone* 
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! learn more when other teaching methods are used. 
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It was a v/aste of time. 
















i 


D 


SD 


7 


SA 


A 




Overall, the course was good. 
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The textbook was very good. 
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The Instructor seemed to oe interested in students as persons. 
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More courses should be taught this way. 
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The course held my interest. 
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1 would have preferred another method of teaching in this course. 
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It was easy to remain attentive. 
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The instructor did not synthesize, integrate or summarize effectively. 
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Not much was gained by taking this course. 
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The instructor encouraged the development of new viewpoints and appreciations. 






WITH THE 


ITEM 










13 


5A 


A 


D 


s r.) 


The course material seemed worthwhile. 
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It was difficult to remain attentive. 
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Instructor did not review promptly and in such a way that students could understand their weaknesses. 
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Homework assignments were helpful in understanding the course. 
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There v;as not enough student participation for this type pf course. 


Is' 


-™ 


A D 
L.... 




The instructor had a thorough knowledge of his subject matter. 
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The content of the course was good. 
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The course inc/eased my general knowledge., 
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The types or test questions used were good. 




COMPLETE SECTIONS BELOW ACCORDING 
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Held my attention throughout the course^ 




TO YOUR INSTRUCTOR'S DIRECTIONS: 
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The demands of the students were not considered by the instructor. 
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Uninteresting course. 
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It was a very worthwhile course* 
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Some things were not explained very well. 
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The way in which this course was taught results in better student learning. 
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The course material was too difficult. 
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One of my poorest courses. 
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Material in the course v/as easy to follow. 
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The instructor seemed to consider teaching as a chore or routine activity. 
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More outside reading is necessary. 
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Course material v/as, poorly organized. 
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Course was not very helpful. 
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It was quite interesting. 
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1 think that the course was taught quite well. 
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\ would prefor a differ eiht mey»od of instruction. 
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The pace of tn*i'courjio was too slow. 
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At times 1 was confused. 
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Excellent course content. 
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The examinations were too difficult. 
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Generally, the course was well organized. 
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Ideas and concepts were developed too rapidly. 


58 


SA 


A 


D 


5D 




93 


SA 


A 


D 


SO 


44 


SA 




D 


SD 


The content of the course was too elementary. 
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Somo days \ was not very interested in this course* 
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It was quite boring. 
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The instructor exhibited profo^-sidnal dignity and bearing in the classroom. 
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Another method of instruction should have been employed. 
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The course was quite useful. 
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1 would take another course that was taught this way. 
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